Phase 4: recovery with confirmation - #10

Phase 4 of the recovery-layer plan: recovery with confirmation.

What changes

The hub (helper 2.4)

• Recoveries. A reset or force off now goes proposed → confirmed by the owner → run → checked. Each one is kept in the hub's state file. States: proposed, running, back, not_back, off, done, failed, declined, withdrawn, expired.
• The hub proposes a reset when a machine turns hung, once per hang.
• Safety rules:
  • At most one recovery runs per machine per 30 minutes. The owner's direct Reset and Force off buttons count towards this, but are never held back.
  • Proposals are withdrawn when the machine answers or goes off, and expire after 6 hours.
  • A run goes through the same gate as the buttons, so it presses nothing if the machine came back.
  • A run cut short by a hub restart ends failed.
• Auto-reset (autoResetMinutes, off by default): the hub confirms its own reset proposal after N minutes while the machine is still hung. Never for force off.
• API:
  • POST /v2/machines/{name}/recoveries needs the power scope. Agents are also subject to the machine's Allow AI agents switch.
  • GET /v2/recoveries[/{id}].
  • POST …/confirm is for owner devices only. Agent keys and an owner hub relaying for an agent are refused.
  • POST …/decline is for the owner, or the key that proposed it.
  • recovery events on SSE and webhooks; webhooks get them by default.
  • Machine.recovery, and hub.autoResetMinutes in status.
  • A direct reset by an agent now points to the propose route.

App

• Recovery card on the device page:
  • Reset and Not now while proposed;
  • "Resetting…" while it runs;
  • after a reset that didn't bring it back, "didn't come back" with Force off.
  • The plain Reset and Restart buttons give way to the card.
• Notification for a proposal: Reset and Not now work only on an unlocked phone (setAuthenticationRequired). It replaces the plain "looks frozen" notification for that machine. Back, not back, failed, and auto or other-device confirms are notified too.
• Setting: the hub's page → Frozen machines → "Reset a frozen machine on its own" (Off, 10, 30 or 60 minutes).

Docs: a Recoveries section in docs/protocol.md, and the CHANGELOG entry for 2.4.0.

Tests

Hub state machine (fake clock, fake NanoKVM):

• the hub proposes, the owner confirms, and the machine is back;
• every propose refusal;
• decline, withdraw and expiry;
• a confirm after the machine came back presses nothing;
• a double confirm runs once;
• not_back, then off-and-on when the reset switch isn't wired;
• auto-reset only for reset, and only while the machine is hung;
• direct buttons are recorded;
• a restart during a run ends failed.

Mutating 8 of the safety rules each makes a test fail.

Server:

• scopes and owner-only confirm (agent key, owner hub relaying for an agent, another account's key);
• too_soon with retryAt;
• agents_off;
• the auto-reset setting's range and that agents can't change it;
• SSE recovery events and the snapshot;
• webhook default events.

Android:

• parsing a recovery and the hub's status;
• card text, with an agent's reason quoted;
• the log-line rules for notifications, with no duplicate "looks frozen";
• Robolectric screenshots of the card (proposed, running, not back) and of the hub setting.

🤖 Generated with Claude Code